Stable Diffusion#

Note

Install ekorpkit package first.

Set logging level to Warning, if you don’t want to see verbose logging.

If you run this notebook in Colab, set Hardware accelerator to GPU.

!pip install ekorpkit[art] exit()

Prepare Environment#

%config InlineBackend.figure_format='retina'
%load_ext autotime

from ekorpkit import eKonf

eKonf.setLogger("WARNING")
eKonf.set_cuda(device=0)
print("version:", eKonf.__version__)

is_colab = eKonf.is_colab()
print("is colab?", is_colab)
if is_colab:
    eKonf.mount_google_drive()
project_dir = eKonf.set_workspace(workspace="/content/drive/MyDrive/workspace/", project="ekorpkit-book")
print("project_dir:", project_dir)
INFO:ekorpkit.base:Setting cuda device to ['A100-SXM4-40GB (id:0)']
INFO:ekorpkit.base:Setting EKORPKIT_WORKSPACE_ROOT to /content/drive/MyDrive/workspace/
INFO:ekorpkit.base:Setting EKORPKIT_PROJECT to ekorpkit-book
INFO:ekorpkit.base:Loaded .env from /workspace/projects/ekorpkit-book/config/.env
The autotime extension is already loaded. To reload it, use:
  %reload_ext autotime
version: 0.1.40.post0.dev8
is colab? False
project_dir: /content/drive/MyDrive/workspace/projects/ekorpkit-book
time: 29.4 ms (started: 2022-11-09 02:31:33 +00:00)

Create a stable diffusion instance#

from ekorpkit.models.art.stable import StableDiffusion

sd = StableDiffusion()
INFO:ekorpkit.base:Loaded .env from /workspace/projects/ekorpkit-book/config/.env
time: 5.6 s (started: 2022-11-09 02:23:16 +00:00)

Generate images#

eKonf.setLogger("INFO")
sd.verbose = True

text_prompts = "a living tiger inside a gorgeous cave in a futuristic Jeju island, relaxing night, watercolour on the ceiling, glowing on glass, light art, dark, dark, gorgeous, dark, bright in an abandoned village in a futuristic alien nightscape"
batch_name = "cave-jeju"
sd.config.collage.ncols = 2

results = sd.imagine(
    text_prompts, 
    batch_name=batch_name, 
    num_samples=4, 
    guidance_scale=9,
    width=800,
    height=400,
    num_inference_steps=150,
)
../../../_images/stable-diffusion_8_0.png
INFO:ekorpkit.models.art.stable: >> elapsed time to imagine: 0:01:44.697041
time: 1min 44s (started: 2022-11-09 02:31:39 +00:00)

Inpaint images#

base_url = "https://github.com/entelecheia/ekorpkit-book/raw/main/assets/figs"
init_image_path = f"{base_url}/chu-horse.png"
mask_image_path = f"{base_url}/chu-horse-masking-rough.png"
mask_image_exact_path = f"{base_url}/chu-horse-masking-exact.png"

init_image = eKonf.load_image(init_image_path)
mask_image = eKonf.load_image(mask_image_path)
mask_image_exact = eKonf.load_image(mask_image_exact_path)

sd.collage([mask_image, init_image, mask_image_exact])
Prompt: a living tiger inside a gorgeous cave in a futuristic Jeju island, relaxing night, watercolour on the ceiling, glowing on glass, light art, dark, dark, gorgeous, dark, bright in an abandoned village in a futuristic alien nightscape
../../../_images/stable-diffusion_10_1.png
time: 3.6 s (started: 2022-11-09 02:25:14 +00:00)
prompt = "a golden statue of an eagle with clouds, colorful painting by Tanya Hern, mural on the roof, digital art, artwork, beautiful, colorful, visual art on landscape, surrealism, watercolor, vivid by stunning"
batch_name = "inpaint-halla"
sd.config.collage.ncols = 3

results = sd.imagine(
    prompt, 
    batch_name=batch_name,
    mode="inpaint",
    init_image=init_image_path,
    mask_image=mask_image_path,
    num_samples=3,
    num_images_per_prompt=1,
    inpaint_strength=10, 
    num_inference_steps=100,
)
Prompt: a golden statue of an eagle with clouds, colorful painting by Tanya Hern, mural on the roof, digital art, artwork, beautiful, colorful, visual art on landscape, surrealism, watercolor, vivid by stunning
../../../_images/stable-diffusion_11_1.png
time: 1min 15s (started: 2022-11-09 02:25:19 +00:00)

Stitch images#

text_prompts = "a living tiger inside a gorgeous cave in a futuristic Jeju island, relaxing night, watercolour on the ceiling, glowing on glass, light art, dark, dark, gorgeous, dark, bright in an abandoned village in a futuristic alien nightscape"
batch_name = "cave-stitch"

results = sd.imagine(
    text_prompts, 
    mode="stitch",
    batch_name=batch_name, 
    num_samples=6, 
    num_images_per_prompt=2,
    guidance_scale=7,
    inpaint_strength=10, 
)
../../../_images/stable-diffusion_13_0.png
INFO:ekorpkit.models.art.stable: >> elapsed time to imagine: 0:01:46.545911
time: 1min 46s (started: 2022-11-09 02:27:08 +00:00)

To view the panorama, upload the image to renderstuff.

Collage Images#

sd.collage(batch_name="lecture", batch_num=9, ncols=2, collage_width=800)
Prompt: examining documents and detailed white background, elegant detailed digital painting, concept art, matte, sharp focus, 4k ultra hd, illustration, magical, unreal engine, octane render, cinematic lighting, wide angle establishing shot, extremely high detail
../../../_images/stable-diffusion_16_1.png
time: 1.99 s (started: 2022-11-08 09:54:01 +00:00)

Show Configuration#

sd.show_config(batch_name="lecture", batch_num=12)
INFO:ekorpkit.batch:Using existing path: /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen
INFO:ekorpkit.batch:Merging config with args: {}
INFO:ekorpkit.batch:Batch name: lecture, Batch num: 12
{'_target_': 'ekorpkit.models.art.stable.StableDiffusion',
 'autoload': True,
 'batch': {'batch_name': 'lecture',
           'batch_num': 12,
           'batch_run_pairs': None,
           'batch_run_params': None,
           'clear_output': True,
           'display_collage': True,
           'max_display_image_width': 1000,
           'num_images_per_prompt': 1,
           'num_samples': 1,
           'random_seed': True,
           'resume_latest': False,
           'resume_run': False,
           'run_to_resume': 'latest',
           'save_collage': True,
           'seed': 2175210533,
           'set_seed': 'random_seed'},
 'collage': {'bg_color': 'black',
             'clear_output': True,
             'collage_width': 1200,
             'cols': -1,
             'crop_to_min_size': False,
             'fontsize': 12,
             'max_images': 6,
             'ncols': 3,
             'padding': 5,
             'show_filename': False,
             'show_prompt': True},
 'device': 'cuda:7',
 'hf_user_access_token': None,
 'imagine': {'clear_output': True,
             'display_collage': True,
             'display_image': True,
             'guidance_scale': 7,
             'height': 400,
             'increase_seed_by': 1,
             'init_image': None,
             'inpaint_strength': 10,
             'make_collage': True,
             'mask_image': None,
             'mode': 'stitch',
             'num_images_per_prompt': 2,
             'num_inference_steps': 150,
             'num_samples': 6,
             'save_collage': True,
             'save_image': True,
             'save_image_config': True,
             'seed': None,
             'text_prompts': 'a living tiger inside a gorgeous cave in a '
                             'futuristic Jeju island, relaxing night, '
                             'watercolour on the ceiling, glowing on glass, '
                             'light art, dark, dark, gorgeous, dark, bright in '
                             'an abandoned village in a futuristic alien '
                             'nightscape',
             'width': 800},
 'model': {'generate': {'device': 'cuda:6',
                        'name': 'CompVis/stable-diffusion-v1-4',
                        'pipeline': 'StableDiffusionPipeline',
                        'pretrained_model_name_or_path': 'CompVis/stable-diffusion-v1-4',
                        'revision': 'fp16',
                        'use_auth_token': True},
           'inpaint': {'device': 'cuda:5',
                       'name': 'runwayml/stable-diffusion-inpainting',
                       'pipeline': 'StableDiffusionInpaintPipeline',
                       'pretrained_model_name_or_path': 'runwayml/stable-diffusion-inpainting',
                       'revision': 'fp16',
                       'use_auth_token': True}},
 'module': {'library_dir': '/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/libs',
            'modules': None},
 'name': 'stable-diffusion',
 'path': {'batch_dir': '/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion/demo',
          'batch_name': 'demo',
          'cache_dir': '/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/cache',
          'library_dir': '/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/libs',
          'model_dir': '/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models',
          'name': 'disco-imagen',
          'output_dir': '/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-diffusion',
          'root': '/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen',
          'tmp_dir': '/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/tmp',
          'verbose': False},
 'verbose': False}
time: 70 ms (started: 2022-11-08 09:58:20 +00:00)